Automated image annotation may be done using neural networks. The neural networks used for image annotation are trained to determine the most appropriate label to apply to a given image. For example, a picture of a cat on a beach may be labeled by a trained neural network with what is considered the most salient feature of the image. If the cat happens to take up most of the image, the cat may be considered the most salient feature, and the neural network may annotate the image with the word “cat.” This may be the result of how the neural network was trained, with images in the training examples used to train the neural network being assigned only one correct label. Thus, a neural network that was trained with the image of the cat on the beach may have been trained that “cat” was the only correct label for the image, and that “beach” was an incorrect label. The neural network may then only apply the label “cat” to similar images, even when a label “beach” might also be considered a correct label for the image. This may limit the usefulness of automated image annotation when preparing images to be searched using a query based search engine.